Papers with Multimodal Transformer
Multiˆ2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open IE systems were based on handcrafted features or fine-grained rules. |
| Approach: | They propose a multi-head argument extraction method using multi-lingual BERT . they use a query, key, and value setting inspired by the Multimodal Transformer . |
| Outcome: | The proposed method outperforms existing sequence-labeling systems on two benchmark datasets and on two languages without training data. |
Multimodal Transformer for Unaligned Multimodal Language Sequences (P19-1)
Copied to clipboard
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
| Challenge: | Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. |
| Approach: | They propose a multimodal model that extends the standard Transformer network to learn representations directly from unaligned multimodal streams. |
| Outcome: | The proposed model outperforms state-of-the-art methods on aligned and non-aligned data. |